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DESCRIPTION 

PLAYBACK APPARATUS, PLAYBACK METHOD, PROGRAM, RECORDING 
MEDIUM, AND DATA STRUCTURE 

Technical Field 
[0001] 

The present invention relates to playback apparatuses, 
playback methods, programs, recording media, and data 
structures. More particularly, the invention relates to a 
playback apparatus, a playback method, a program, a 
recording medium, and a data structure that enable 
interactive operations when AV content is played back. 
Background Art 
[0002] 

In DVD (Digital Versatile Disc) video standards, 
interactive operations can be performed, that is, users can 
switch sound or subtitles when playing back AV (Audio 
Visual) content, such as a movie, recorded on an information 
recording medium (for example, see Non- Patent Document 1) . 
More specifically, the user operates a sound switch button 
11 or a subtitle switch button 12 of a remote controller 2 
to switch sound or subtitles of AV content displayed on a 
display device 1 shown in Fig. 1. For example, if the user 
operates the sound switch button 11 when sound 1 is set in 
the initial state in the display device 1, sound 1 is 
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switched to sound 2, as shown in Fig. 2. 
[0003] 

AV content on DVD video is recorded in the form of an 
MPEG (Moving Picture Experts Group) 2 program stream. In the 
MPEG2 program stream, as shown in Fig. 3, a video stream 

(Video in Fig. 3) , a plurality of audio streams (indicated 
by Audio 1, 2, and 3 in Fig. 3), and a plurality of sub- 
picture streams (Sub-pictures 1, 2, and 3) are multiplexed 
such that the audio streams and the sub-picture streams are 
AV- synchronized with the video stream. The sub-picture 
streams (sub-pictures 1, 2, and 3) are streams in which 
bitmap images are run-length coded, and are mainly used for 
subtitles . 

[0004] 

Generally, a plurality of audio streams are used for 
recording sound of different languages, and a plurality of 
sub-picture streams are used for recording subtitles of 
different languages. The user can interactively select 
sound or subtitles of a desired language by using the remote 
controller 2 while video is being played baclt. 
[0005] 

The DVD video defines a table structure, provided to 
users, indicating the relationship between sound numbers and 
subtitle numbers for a plurality of audio streams (audio 1, 
2, and 3) and a plurality of sub-picture streams (sub- 
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pictures 1, 2, and 3) in a program stream. 
[0006] 

Fig. 4 illustrates a stream number table indicating the 
relationship between audio signals and subtitle signals. In 
this table, the sound numbers are referred to as "A_SN 

(Audio Stream Number) " , and the subtitle numbers are 
referred to as "S_SN (Sub-picture Stream Number)". In Fig. 
4, each of the plurality of audio streams is provided with 
A_SN, and each of the plurality of sub-picture streams is 
provided with S_SN. More specifically, A_SN=1 : audio 2, 
A_SN=2 : audio 1, and A_SN-3: audio 3. Also, S_SN=1 : sub- 
picture 3, S_SN=2 : sub-picture 1, and S_SN=3 : sub-picture 2. 
In this case, a smaller number of A_SN or S_SN indicates an 
audio signal or a subtitle signal to be provided to users 
with higher priority. That is, A_SN=1 is an audio stream 
played back as a default, and S__SN=1 is a sub-picture stream 
played back as a default. 

[0007] 

More specifically, sound 1 played back in the initial 
state in Fig. 1 is audio 2, which is A_SN=1 (Fig. 4), and 
sound 2 played back after being switched from sound 1 in Fig. 
2 is audio 1, which is A_SN=2 . 

Non-Patent Document 1: DVD Specifications for Read-Only Disc 
Part 3; Version 1.1 
Disclosure of Invention 
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Problems to be Solved by the Invention 
[0008] 

According to the DVD video, however, when the user 
switches sound or subtitles while playing back a video 
program stream, the user can select only from audio streams 
or sub-picture streams multiplexed into the program stream 
which is currently played back. That is, when playing back 
an MPEG2 program stream, such as that shown in Fig. 3, the 
user can select only from audio 1 through audio 3 when 
switching sound. 
[0009] 

Accordingly, even if another stream having audio 
streams and subtitles different from a program stream which 
is currently played back is available, the user cannot 
switch sound or subtitles to the audio streams or subtitles 
in the different stream. Accordingly, the extensibility in 
selecting streams is low. 
[0010] 

The present invention has been made in view of the 
above background. An object of the present invention is to 
select sound and subtitles from streams or data files 
different from a main AV stream when the user switches sound 
or subtitles. 

Means for Solving the Problems 
[0011] 
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A playback apparatus of the present invention includes: 
obtaining means for obtaining playback management 
information including first information having a main 
playback path indicating the position of an AV stream file 
recorded on a recording medium and second information having 
a plurality of sub playback paths indicating the positions 
of sub files including accompanying data to be played back 
simultaneously with playback of main image data included in 
the AV stream file; selection means for selecting 
accompanying data to be played back, based on an instruction 
from a user, from among accompanying data to be played back 
simultaneously with the main image data included in the AV 
stream file referred to by the main playback path and the 
accompanying data included in the sub files referred to by 
the sub playback paths; reading means for reading, if the 
accompanying data selected by the selection means is 
included in a sub file referred to by a sub playback path, 
the sub file referred to by the sub playback path together 
with the AV stream file referred to by the main playback 
path; and playback means for playing back the main image 
data included in the AV stream file read by the reading 
means and the accompanying data included in the sub file 
selected by the selection means and read by the reading 
means . 
[0012] 
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The first information may include a table defining the 
accompanying data included in the AV stream file referred to 
by the main playback path and the accompanying data referred 
to by the sub playback paths, and the selection means may 
select the accompanying data to be played back, based on the 
instruction from the user, from among the accompanying data 
defined in the table. 
[0013] 

The playback apparatus may further include determining 
means for determining whether the playback apparatus has a 
function of playing back the accompanying data selected by 
the selection means. If it is determined by the determining 
means that the playback apparatus has a function of playing 
back the accompanying data and if the accompanying data is 
included in a sub file referred to by a sub playback path, 
the reading means may read the sub file referred to by the 
sub playback path together with the AV stream file referred 
to by the main playback path, and the playback means may 
play back the main image data included in the AV stream file 
read by the reading means and the accompanying data included 
in the sub file selected by the selection means and read by 
the reading means . 
[0014] 

The playback apparatus may further include determining 
means for determining whether the playback apparatus has a 
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function of playing back the accompanying data selected by 
the selection means. If it is determined by the determining 
means that the playback apparatus has a function of playing 
back the accompanying data and if the accompanying data is 
included in a sub file referred to by a sub playback path, 
the reading means may read the sub file referred to by the 
sub playback path together with the AV stream file referred 
to by the main playback path, and the playback means may 
play back the main image data included in the AV stream file 
read by the reading means and the accompanying data included 
in the sub file selected by the selection means and read by 
the reading means . 
[0015] 

The table may further define accompanying information 
concerning accompanying data, and the determining means may 
determine whether the playback apparatus has a function of 
playing back the accompanying data based on attribute 
information concerning the accompanying data defined in the 
table . 
[0016] 

The second information may include type information 
concerning the types of sub playback paths, file names of 
the sub files referred to by the sub playback paths, and IN 
points and OUT points of the sub files referred to by the 
sub playback paths. 
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[0017] 

The second information may further include specifying 
information for specifying the AV stream file referred to by 
the main playback path to play back the sub playback paths 
simultaneously with the main playback path, and a time on 
the main playback path for allowing the IN points to be 
started in synchronization with the main playback path on 
the time axis of the main playback path. 
[0018] 

A playback method of the present invention includes: an 
obtaining step of obtaining playback management information 
including first information having a main playback path 
indicating the position of an AV stream file recorded on a 
recording medium and second information having a plurality 
of sub playback paths indicating positions of sub files 
including accompanying data to be played back simultaneously 
with playback of main image data included in the AV stream 
file; a selection step of selecting accompanying data to be 
played back, based on an instruction from a user, from among 
accompanying data to be played back simultaneously with the 
main image data included in the AV stream file referred to 
by the main playback path and the accompanying data included 
in the sub files referred to by the sub playback paths; a 
reading step of reading, if the accompanying data selected 
by processing of the selection step is included in a sub 
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file referred to by a sub playback path, the sub file 
referred to by the sub playback path together with the AV 
stream file referred to by the main playback path; and a 
playback step of playing back the main image data included 
in the AV stream file read by processing of the reading step 
and the accompanying data included in the sub file selected 
by processing of the selection step and read by processing 
of the reading step. 
[0019] 

A program of the present invention includes: an 
obtaining step of obtaining playback management information 
including first information having a main playback path 
indicating the position of an AV stream file recorded on a 
recording medium and second information having a plurality 
of sub playback paths indicating the positions of sub files 
including accompanying data to be played back simultaneously 
with playback of main image data included in the AV stream 
file; a selection step of selecting accompanying data to be 
played back, based on an instruction from a user, from among 
accompanying data to be played back simultaneously with the 
main image data included in the AV stream file referred to 
by the main playback path and the accompanying data included 
in the sub files referred to by the sub playback paths; a 
reading step of reading, if the accompanying data selected 
by processing of the selection step is included in a sub 
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file referred to by a sub playback path, the sub file 
referred to by the sub playback path together with the AV 
stream file referred to by the main playback path; and a 
playback step of playing back the main image data included 
in the AV stream file read by processing of the reading step 
and the accompanying data included in the sub file selected 
by processing of the selection step and read by processing 
of the reading step. 
[0020] 

According to a first aspect of the present invention, 
playback management information including first information 
having a main playback path indicating the position of an AV 
stream file recorded on a recording medium and second 
information having a plurality of sub playback paths 
indicating the positions of sub files including accompanying 
data to be played back simultaneously with playback of main 
image data included in the AV stream file is obtained. Then, 
accompanying data to be played back is selected, based on an 
instruction from a user, from among accompanying data to be 
played back simultaneously with the main image data included 
in the AV stream file referred to by the main playback path 
and the accompanying data included in the sub files referred 
to by the sub playback paths. If the selected accompanying 
data is included in a sub file referred to by a sub playback 
path, the sub file referred to by the sub playback path is 
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read together with the AV stream file referred to by the 
main playback path. Then, main image data included in the 
read AV stream file and the accompanying data included in 
the selected and read sub file are played back. 
[0021] 

Association data recorded on a first recording medium 
of the present invention indicates whether accompanying data 
is included in a clip used by a main playback path 
indicating the position of the AV stream file or in clips 
used by a plurality of sub playback paths indicating the 
positions of sub files including the accompanying data 
played back simultaneously with the AV stream file. If the 
association data indicates that the accompanying data is 
included in the clips used by the plurality of sub playback 
paths indicating the positions of the sub files including 
the accompanying data, the association data includes at 
least an ID of the sub playback path to be played back 
selected from the ID for specifying the sub playback path to 
be played back, an ID for specifying a clip used by the sub 
playback path, and an ID for specifying an elementary stream 
to be played back by the clip. 
[0022] 

According to a second aspect of the present invention, 
the association data indicates whether accompanying data is 
included in a clip used by a main playback path indicating 
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the position of the AV stream file or in clips used by a 
plurality of sub playback paths indicating the positions of 
sub files including the accompanying data played back 
simultaneously with the AV stream file. If the association 
data indicates that the accompanying data is included in the 
clips used by the plurality of sub playback paths indicating 
the positions of the sub files including the accompanying 
data, the association data includes at least an ID of the 
sub playback path to be played back selected from the ID for 
specifying the sub playback path to be played back, an ID 
for specifying a clip used by the sub playback path, and an 
ID for specifying an elementary stream to be played back by 
the clip. 
[0023] 

A playback control file recorded on a second recording 
medium of the present invention includes a sub playback path 
indicating the position of a sub file including accompanying 
data to be played back simultaneously with main image data 
included in an AV stream file. The main playback path 
includes a table defining a list of elementary streams that 
are selectable while the main playback path is being played 
back. The table includes data indicating whether the 
selectable elementary streams are included in the AV stream 
file selected by the main playback path or the sub file 
selected by the sub playback path. 
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[0024] 

According to a data structure of the present invention, 
a playback control file includes a sub playback path 
indicating the position of a sub file including accompanying 
data to be played back simultaneously with main image data 
included in an AV stream file. The main playback path 
includes a table defining a list of elementary streams that 
are selectable while the main playback path is being played 
back. The table includes data indicating whether the 
selectable elementary streams are included in the AV stream 
file selected by the main playback path or the sub file 
selected by the sub playback path. 
[0025] 

According to a third aspect of the present invention, a 
playback control file includes a sub playback path 
indicating the position of a sub file including accompanying 
data to be played back simultaneously with main image data 
included in an AV stream file. The main playback path 
includes a table defining a list of elementary streams that 
are selectable while the main playback path is being played 
back. The table includes data indicating whether the 
selectable elementary streams are included in the AV stream 
file selected by the main playback path or the sub file 
selected by the sub playback path. 
Advantages 
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[0026] 

According to the present invention, interactive 
operations can be performed when playing back an AV stream 
file. In particular, according to the present invention, 
interactive operations can also be performed on sub files 
referred to by sub playback paths, which are different from 
the AV stream file referred to by the main playback path. 
Brief Description of the Drawings 
[0027] 

[Fig. 1] Fig. 1 illustrates known sound switching. 

[Fig. 2] Fig. 2 illustrates known sound switching. 

[Fig. 3] Fig. 3 illustrates the structure of an MPEG2 
program stream. 

[Fig. 4] Fig. 4 illustrates a stream number table 
indicating the relationship between sound signals and 
subtitle signals provided to a user. 

[Fig. 5] Fig. 5 illustrates an example of an 
application format on a recording medium installed in a 
playback apparatus to which the present invention is applied. 

[Fig. 6] Fig. 6 illustrates the structure of Main Path 
and Sub Path. 

[Fig. 7] Fig. 7 illustrates an example of Main Path 
and Sub Path. 

[Fig. 8] Fig. 8 illustrates another example of Main 
Path and Sub Path. 
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[Fig. 9] Fig. 9 illustrates still another example of 
Main Path and Sub Path. 

[Fig. 10] Fig. 10 illustrates another example of Main 
Path and Sub Path. 

[Fig. 11] Fig. 11 illustrates the syntax of 
PlayList () . 

[Fig. 12] Fig. 12 illustrates the syntax of SubPathO . 

[Fig. 13] Fig. 13 illustrates the syntax of 
SubPlayltem (i) . 

[Fig. 14] Fig. 14 illustrates the syntax of 
PlayltemO . 

[Fig. 15] Fig. 15 illustrates the syntax of 
STN_table 0 . 

[Fig. 16] Fig. 16 illustrates an example of the syntax 
of stream_entry ( ) . 

[Fig. 17] Fig. 17 illustrates the syntax of 
stream attribute () . 
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illustrates 


stream^cording_type . 
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Fig . 


19 


illustrates 


video_f ormat . 
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Fig. 


20 


illustrates 


f rame_rate . 


[Fig. 


.21] 


Fig. 


21 


illustrates 


aspect_ratio . 


[Fig. 
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Fig. 


22 


illustrates 


audio_presentation_type . 


[Fig. 
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Fig. 


23 


illustrates 


sampl ing_f requency . 


[Fig. 
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Fig . 


24 


illustrates 


Character code . 


[Fig. 


25] 


Fig. 


25 


is a block diagram illustrating an 
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example of the configuration of a playback apparatus to 
which the present invention is applied. 

[Fig. 26] Fig. 26 is a flowchart illustrating playback 
processing performed by the playback apparatus shown in Fig. 
25. 

[Fig. 27] Fig. 27 is a flowchart illustrating the 
playback processing performed by the playback apparatus 
shown in Fig. 25. 

[Fig. 28] Fig. 28 is a flowchart illustrating the 
playback processing performed by the playback apparatus 
shown in Fig. 25. 

[Fig. 29] Fig. 29 is a flowchart illustrating 
processing when an instruction to switch audio is given by a 
user. 

[Fig. 30] Fig. 30 is a flowchart illustrating 
processing when an instruction to switch subtitles is given 
by a user. 

[Fig. 31] Fig. 31 illustrates the configuration of a 
personal computer. 

[Fig. 32A] Fig. 32A illustrates another example of the 
syntax of PlayListO . 

[Fig. 32B] Fig. 32B illustrates another example of the 
syntax of PlayListO . 

[Fig. 33] Fig. 33 illustrates another example of the 
syntax of STN_table() . 
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[Fig. 34] Fig. 34 illustrates the types in STN_table() 
shown in Fig. 33. 
Reference Numerals 
[0028] 

2 0 playback apparatus, 31 storage drive, 32 switch, 
33 AV decoder, 34 controller, 51 through 54 buffers, 55, 
56 PID filters, 57 through 59 switches, 71 baclcground 
decoder, 72 MPEG2 video decoder, 73 presentation graphics 
decoder, 74 interactive graphics decoder, 75 audio decoder, 
76 Text -ST composition, 77 switch, 91 baclcground plane 
generator, 92 video plane generator, 93 presentation 
graphics plane generator, 94 interactive graphics plane 
generator, 95 buffer, 96 video data processor, 97 audio 
data processor 

Best Mode for Carrying Out the Invention 
[0029] 

An embodiment of the present invention is described 
below with reference to the accompanying drawings. 
[0030] 

Fig. 5 illustrates an example of an application format 
on a recording medium to be installed in a playbaclc 
apparatus 20 (which is discussed below with reference to Fig. 
25) to which the present invention is applied. The 
recording medium may be an optical disc, a magnetic dislc, or 
a semiconductor memory, which is discussed below. 
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[0031] 

The application format has two layers, i.e., PlayList 
and Clip, for managing AV (Audio Visual) streams. In this 
case, a pair of one AV stream and one item of clip 
information accompanying the AV stream are considered to be 
as one object, which is referred to as a "clip". An AV 
stream is also referred to as an "AV stream file". Clip 
information is also referred to as a "clip information file". 
[0032] 

Generally, files used in computers are handled as byte 
strings. On the other hand, content of AV stream files is 
expanded onto a time axis, and access points in clips are 
mainly specified by PlayLists by using time stamps. That is, 
it can be said that PlayList and Clip form two layers for 
managing AV streams . 
[0033] 

If access points in Clip are indicated by PlayList by 
using time stamps, a clip information file is used for 
finding, from the time stamps, information concerning an 
address at which decoding in an AV stream file is started. 
[0034] 

PlayList is a set of playback zones of an AV stream. 
One playback zone in an AV stream is referred to as 
"Playltem", which is indicated by a pair of an IN point 
(playback start point) and an OUT point (playback end point) 
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on the time axis. Accordingly, PlayList has one or a 
plurality of Playl terns , as shown in Fig. 5. 
[0035] 

In Fig. 5, the first PlayList from the left has two 
Playltems, which refer to the first half and the second half 
of the AV stream contained in the clip at the left side in 
Fig. 5. The second PlayList from the left has one Playltem, 
which refers to the entirety of the AV stream contained in 
the clip at the right side. The third PlayList from the 
left has two Playltems, which refer to a certain portion of 
the AV stream contained in the clip at the left side and a 
certain portion of the AV stream contained in the clip at 
the right side. 
[0036] 

If a disc navigation program shown in Fig. 5 designates 
the left Playltem contained in the first PlayList from the 
left as information concerning the current playback position, 
the first half of the AV stream contained in the left clip, 
which is referred to by the designated Playltem, is played 
back. 
[0037] 

The disc navigation program has a function of 
controlling the playback order specified in PlayList and 
interactive playback operations by using PlayList. The disc 
navigation program also has a function of displaying a menu 
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screen for allowing a user to give instructions to perform 
various types of playback operations. The disc navigation 
program is described in a programming language, for example, 
Java™, and is recorded on a recording medium. 
[0038] 

In this embodiment/ a playback path including at least 
one Playltem (sequential Playltems) in PlayList is referred 
to as a "Main Path" , and a playback path including at least 
one Sub Path (formed by sequential or non- sequential 
SubPlayltems) disposed in parallel with the Main Path in a 
PlayList is referred to as a "Sub Path". That is, the 
application format on a recording medium installed in the 
playback apparatus 2 0 (which is discussed below with 
reference to Fig. 25) has at least one Sub Path, which is 
played back in association with the Main Path, in a PlayList. 
[0039] 

Fig. 6 illustrates the structure of the Main Path and 
Sub Path. The PlayList is allowed to have the single Main 
Path and at least one Sub Path. One Sub Path includes at 
least one SubPlayltem. 
[0040] 

The PlayList shown in Fig. 6 has one Main Path 
including three Playltems and three Sub Paths. The 
Playltems forming the Main Path are provided with IDs 
(Identifications) from the head. More specifically, the 
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Main Path includes Playltems, such as PlayItem_id=0 , 
Playltem_id=l, and Playltem_id=2 . The Sub Path are also 
provided with IDs from the head, such as Subpath_id=0 , 
Subpath_id=l, and Sub Path_id=2 . Subpath_id=0 has one 
SubPlayltem, Subpath_id=l has two SubPlayl terns , and Sub 
Path__id=2 has one SubPlayltem, 
[0041] 

Sub Path_id=l is applied to, for example. Director's 
Cut, and can be inserted into a predetermined portion of the 
AV stream file as the director's commentaries. 
[0042] 

A clip AV stream file referred to by one Playltem 
includes at least video stream data (main image data) . The 
clip AV stream file may also include at least one audio 
stream, which is played back simultaneously with (in 
synchronization with) the video stream (main image data) 
which is also contained in the clip AV stream file. The 
clip AV stream file may also include at least one bitmap 
subtitle stream file which is played back in synchronization 
with the video stream which is also contained in the clip AV 
stream file. The clip AV stream file may also include at 
least one interactive graphics stream file which is played 
back in synchronization with the video stream which is also 
contained in the clip AV stream file. The video stream 
contained in the clip AV stream file and the audio stream. 
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bitmap subtitle stream, or interactive graphics stream, 
which is played back in synchronization with the video 
stream, are multiplexed- In other words, a clip AV stream 
file referred to by one Playltem includes video stream data 
and at least 0 audio stream data, at least 0 bitmap subtitle 
stream data, or at least 0 interactive graphics stream data, 
which is played back in synchronization with the video 
stream data, such that they are multiplexed into the clip AV 
stream file. 
[0043] 

A SubPlayltem refers to, for example, audio stream data 
or subtitle data contained in a stream different from the 
clip AV stream file referred to by the Playltem. 
[0044] 

When playing back a PlayList including only a Main Path, 
the user can select sound and subtitles only from audio 
streams and sub-picture streams multiplexed into a clip 
referred to by that Main Path. In contrast, when playing 
back a PlayList including a Main Path and a Sub Path, the 
user can refer to audio streams and sub-picture streams 
multiplexed into a clip referred to by the SubPlayltem in 
addition to audio streams and sub-picture streams 
multiplexed into a clip AV stream file referred to by the 
Main Path. 
[0045] 
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As discussed above, a plurality of Sub Paths are 
included in one PlayList, and each SubPath refers to the 
corresponding SubPlayltem. Accordingly, AV streams having 
high extensibility and high flexibility can be provided. 
That is, SubPlayl terns can be added afterwards. 
[0046] 

Fig. 7 illustrates an example of the Main Path and an 
example of the Sub Path. In Fig. 7, an audio playback path 
played back simultaneously with (in AV- synchronization with) 
the Main Path is indicated by using the Sub Path. 
[0047] 

The PlayList shown in Fig. 7 includes one Playltem, 
i.e., PlayItem_id=0, as the Main Path, and one SubPlayltem 
as the Sub Path. The SubPlayltem ( ) includes the following 
data. SubPlayltem 0 includes Clip_Inf ormation_f ile_name for 
specifying the clip referred to by the Sub Path in the 
PlayList. In the example in Fig. 7, the SubPlayltem refers 
to an Auxiliary audio stream of SubClip_entry_id=0 . 
SubPlayltem 0 also includes SubPlayItem_IN_time and 
SubPlayl tem_OUT_time for specifying the playback zone of the 
Sub Path contained in the clip (in this case, the Auxiliary 
audio stream) . The SubPlayltem ( ) also includes 
sync_PlayItem_id and sync_start_PTS_of _PlayItem for 
specifying the time at which the playback operation of the 
Sub Path is started on the time axis of the Main Path. In 
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Fig. 7, sync_PlayItem_id=0 and sync_start_PTS_of __PlayItem=tl . 
With this information, the time tl at which the playback 
operation of the Sub Path is started on the time axis of 
PlayItem_id=0 of the Main Path can be specified. That is, 
in the example in Fig. 7, the playback start time of the 
Main Path and the playback start time of the Sub Path is the 
same , i.e., t 1 . 
[0048] 

The clip AV audio stream referred to by the Sub Path 
should not include STC non- sequential points (system time 
base non- sequential points) . The clip audio sample clock 
used for the Sub Path is locked to the audio sample clock 
used for the Main Path. 
[0049] 

In other words, SubPlayltem ( ) includes information for 
specifying the clip referred to by the Sub Path, information 
for specifying the playback zone of the Sub Path, and 
information for specifying the time at which the playback 
operation of the Sub Path is started on the time axis of the 
Main Path. Since the clip AV stream used for the Sub Path 
does not include STC, the user can refer to a clip AV audio 
stream different from the clip AV stream (main AV stream) 
referred to by the Main Path on the basis of the information 
included in SubPlayltem ( ) (information for specifying the 
clip referred to by the Sub Path, information for specifying 
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the playback zone of the Sub Path, and information for 
specifying the time at which the playback operation of the 
Sub Path is started on the time axis of the Main Path) , and 
plays back the clip AV audio stream. 
[0050] 

As stated above, the Playltem and the SubPlayltem 
individually manage clip AV stream files. The clip AV 
stream file managed by the Playltem is different from the 
clip AV stream file managed by the SubPlayltem. 
[0051] 

In a manner similar to the example shown in Fig. 1 , a 
subtitle stream playback path played back simultaneously 
with the Main Path may be indicated by using a Sub Path. 
[0052] 

Fig. 8 illustrates another example of Main Path and 
another example of Sub Path. In Fig. 8, an audio playback 
path played back asynchronously with the Main Path is 
indicated by using a Sub Path. The main clip AV stream file 
referred to by the Playltem of the Main Path is similar to 
that in Fig. 7, and an explanation thereof is thus omitted. 
[0053] 

The configuration shown in Fig. 8 is used when, for 
example, the Main Path is used as the slideshow of still 
images and the audio path of the Sub Path is used as the BGM 
(background music) of the Main Path. That is, the 
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configuration shown in Fig. 8 is used for allowing BGM to be 
continuously played back when the user instructs a playback 
apparatus (player) to update the images of the slideshow. 
[0054] 

In Fig. 8, PlayItem__id=0 , 1, and 2 are disposed in the 
Main Path, and one SubPlayltem is disposed in the Sub Path. 
The Sub Path includes SubPlayItem_IN_time and 

SubPlayItem_OUT_time for specifying the playback zone of the 
Sub Path in the clip (Auxiliary audio stream) . In the 
example in Fig. 8, the clip (Auxiliary audio stream) is 
referred to by the SubPlayltem. By comparing Fig. 8 with 
Fig. 7, it can be understood that the SubPlayltem in Fig. 8 
does not include sync_PlayItem_id and 

sync_start_PTS_of_PlayItem. The reason for this is that, 
since the playback timing of the AV stream (video data) 
referred to by the Main Path is not related to the audio 
playback timing, it is not necessary to specify the time at 
which the Sub Path starts the playback operation on the time 
axis of the Main Path. That is, information indicating that 
the AV stream referred to by the Main Path and the audio 
stream referred to by the Sub Path are played back together 
is sufficient. 
[0055] 

It has been described that the playback timing of the 
video stream data included in the AV stream is different 
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from that of the audio stream referred to by the Sub Path. 
To put it more specifically, this means that the playback 
timing of the video stream included in the AV stream is 
synchronous with that of the video stream (i.e., the audio 
stream is associated with the video stream) , but a specific 
association, i.e., while a predetermined frame in the video 
stream is being played back, the corresponding sound is 
played back, is not given. More specifically, in the 
example shown in Fig. 7, the playback timing of the video 
stream is synchronous with that of the audio stream, and 
also, a specific association, i.e., while a predetermined 
frame in the video stream is being played back, the 
corresponding sound is played back, is given. In contrast, 
in the example shown in Fig. 8, although the playback timing 
of the video stream is synchronous with that of the audio 
stream, a specific association is not given, which means 
that, while a predetermined frame in the video stream is 
being played back, the corresponding sound is not played 
back. 
[0056] 

Fig. 9 illustrates another example of the Main Path and 
the Sub Path. In Fig. 9, the playback path of text 
subtitles (interactive graphics stream) played back 
simultaneously with the Main Path is indicated by using a 
Sub Path. The main AV stream file referred to by the 
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Playltem of the Main Path is similar to that shown in Fig. 7, 
and an explanation thereof is thus omitted. 
[0057] 

In this case, the text subtitles are defined as a 
multiplexed stream of an MPEG-2 system or as a data file, 
which is not a multiplexed stream. The data file is a file 
containing the text data of subtitles (character code 
string) to be played back in synchronization with the video 
of the Main Path and the attributes of the text data. The 
attributes are information concerning the font type, font 
size, and character color when the text data is subjected to 
rendering . 
[0058] 

By comparing Fig. 9 with Fig. 7, it can be understood 
that the SubPlayltem can refer to the Text based subtitle 
(text subtitles) of SubClip_entry_id=0 , 1, . . . , N by the 
SubPlayltem. More specifically, according to the structure 
shown in Fig. 9, a plurality of text subtitle files can be 
simultaneously referred to by one SubPlayltem, and when 
playing baclc the SubPlayltem, one of the plurality of text 
subtitle files is selected and played bacJc. For example, 
from among the text subtitles of a plurality of languages, 
one text subtitle file is selected and played back. More 
specifically, one SubClip_entry__id is selected from 
SubClip__entry_id=0 through N (based on a user's instruction). 
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and the Text based subtitle referred to by the ID of the 
selected ID is played back. 
[0059] 

Not only the text subtitle files, bitmap subtitle 
stream files, transport stream files, and various data files 
may be applicable to the example show in Fig. 9. 
Alternatively, data files including character codes and 
information for rendering the character codes may also 
applicable . 
[0060] 

Fig. 10 illustrates another example of the Main Path 
and the Sub Path. In Fig. 10, the playbaclc path of an 
interactive graphics stream played back asynchronously with 
the Main Path is indicated by using a Sub Path. 
[0061] 

By comparing Fig. 10 with Fig. 8, it can be understood 
that the interactive graphics stream of SubClip_entry_id=0 , 
1, N can be referred to by the SubPlayltem. That is, 

according to the structure in Fig. 10, one SubPlayltem can 
simultaneously refer to a plurality of interactive graphics 
stream files. When playing back the SubPlayltem, one 
interactive graphics stream file is selected and played back 
from the plurality of interactive graphics stream files. 
More specifically, from among SubClip_entry_id=0 through N, 
one SubClip_entry_id is selected (based on a user's 
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instruction) , and the interactive graphics stream referred 
to by the ID is played back. For example, based on the 
user's instruction, one language from the interactive 
graphics streams is selected, and the interactive graphics 
stream of the selected language is played back. 
[0062] 

The data structure (syntax) that implements the 
structure of the Main Path and Sub Paths discussed with 
reference to Figs. 6 through 10 is described below. 
[0063] 

Fig. 11 illustrates the syntax of PlayListO. 
[0064] 

The "length" is a 32 -bit unsigned integer indicating 
the number of bytes from immediately after the length field 
to the end of PlayListO, i.e., a field indicating the 
number of bytes from reserved_f or_f uture__use to the end of 
PlayListO. After the "length", the 16-bit 

reserved_f or_future_use follows . The "number_of_PlayItems" 
is a 16-bit field indicating the number of play items 
contained in the PlayList . In the case of the example in 
Fig. 6, the number of Playltems is three, and the numeric 
value is assigned to the Playltems as Playltem_id from 0 in 
the order in which PlayltemO appears in the playlist. For 
example, PlayItem_id=0 , 1, 2 are assigned, as shown in Fig. 
6, 8, or 10. 



- 31 - 

S05P0181 

[0065] 

The number_of_SubPaths is a 16-bit field indicating the 
number of Sub Paths (number of entries) contained in the 
PlayList. In the case of the example in Fig, 6, the number 
of Sub Paths is three, and the numeric value is assigned to 
the SubPlayltems as SubPath^id from 0 in the order in which 
SubPathO appears in the playlist. For example. Sub 
Path_id=0, 1, 2 are assigned, as shown in Fig. 6. Then, in 
the subsequent FOR statement, the Playltems are referred to 
for the same number of times as the number of Playltems, and 
the Sub Paths are referred to for the same number of times 
as the number of Sub Paths . 
[0066] 

As an alternative to the syntax of PlayList () shown in 
Fig. 11, the Syntax shown in Fig. 32 can be considered. In 
Fig. 11, the data structure SubPathO storing the 
information concerning the SubPath is contained in the 
PlayList (). However, in Fig. 32, the data structure 
SubPathO is disposed independently of PlayList (). In 
PlayList 0 shown in Fig. 32A, only Playltems of Main Path 
are indicated, and in SubPathsO shown in Fig. 32B, Sub Path 
and SubPlayltems are indicated. According to the data 
structure shown in Fig. 32, SubPathsO can be stored in a 
file different from the file storing PlayList (). For 
example, the file storing SubPathO and a subtitle stream 
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file or an audio stream file referred to by the SubPath can 
be downloaded from a network and can be played back together 
with the Main Path stored in a recording medium. That is, 
the extensibility of SubPath can be easily implemented. The 
file storing PlayListO and the file storing SubPaths ( ) can 
be associated with each other by, for example, allowing part 
of the file names of the two files to be the same. 
[0067] 

Fig. 12 illustrates the syntax of SubPath () . 
[0068] 

The "length" is a 32 -bit unsigned integer indicating 
the number of bytes from immediately after the length field 
to the end of SubPath (). After the "length", 16 -bit 
reserved_for_future_use follows. The SubPath_type is an 8- 
bit field indicating the type of application of the SubPath. 
The SubPath_type is used for indicating, for example, the 
type of Sub Path, such as audio, bitmap subtitle, or text 
subtitle. That is, the SubPath^type indicates the types of 
Sub Paths shown in Figs. 7 through 10. After the 
SubPath_type, 15-bit reserved_f or__f uture_use follows. The 
is_repeat_SubPath is a one-bit field indicating the playback 
operation for the SubPath, and more specifically, indicating 
whether the SubPath is played back repeatedly or only once 
while playing back the Main Path. This field is used when, 
for example, the playback timing of the stream contained in 
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the clip specified by the Sub Path is different from that of 
the main AV stream, as shown in Fig. 8 or 10. After 
is_repeat_SubPath, 8-bit reserved_f or_f uture_use follows . 
The number_of_SubPlayItems is an 8 -bit field indicating the 
number of SubPlayl terns (number of entries) contained in one 
SubPath. For example, the number of SubPlayl terns of 
SubPath_id=0 in Fig. 6 is one, and the number of 
SubPlayltems of Sub Path_id=l is 2 . In the subsequent FOR 
statement, SubPlayltems are referred to for the same number 
of times as the number of SubPlayltems. 
[0069] 

Fig. 13 illustrates the syntax of SubPlayl tem (i) . 
[0070] 

The "length" is a 16 -bit unsigned integer indicating 
the number of bytes from immediately after the length field 
to the end of SubPlayltem ( ) . 
[0071] 

In Fig. 13, the syntax is divided into two portions, 
and more specifically, a portion where the SubPlayltem 
refers to one clip and a portion where the SubPlayltem 
refers to a plurality of clips are shown. 
[0072] 

The portion where the SubPlayltem refers to one clip is 
discussed first. 
[0073] 
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The SubPlayltemO includes 
Clip_Inf orTnation_f ile_name [0] for specifying the clip. The 
SubPlayltemO also includes Clip_codec_identif ier [0] for 
specifying the codec method for the clip, 

reserved_f or_future_use, is_multi_Clip_entries, which is a 
flag indicating whether multi -clips are registered, and 
ref_to_STC__id [0] , which is information concerning the STC 
non- sequential points (non- sequential points of the system 
time base) • If the flag of is_multi_Clip_entries is ON, the 
syntax of the portion where SubPlayltemO refers to a 
plurality of clips is checked. The SubPlayltemO also 
includes SubPlayItem_IN_time and SubPlayItem__OUT_time for 
specifying the playback zone of the Sub Path contained in 
the clip, and sync__PlayItem_id and 

sync__start_PTS_of_PlayItem for specifying the playback start 
time at which the playback operation of the Sub Path is 
started on the time axis of the Main Path. The 
sync_PlayItem_id and sync_start_PTS_of _PlayItem are used 
when the playback timing of the main AV stream is the same 
as that of the stream contained in the file referred to by 
the Sub Path, as shown in Figs. 7 and 9, but are not used 
for the cases shown in Figs. 8 and 10 (when the playback 
timing of the main AV stream is different from the stream 
contained in the file referred to by the Sub Path) . The 
SubPlayItem_IN_time, SubPlayItem_OUT_time , sync_PlayItem_id, 
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and sync__start_PTS_of_PlayItem are used in common for the 
clip referred to by SubPlayltem. 
[0074] 

Next, the portion where the SubPlayltem refers to a 
plurality of clips (if (is_multi__Clip_entries = = lb) is 
discussed. More specifically, the case where SubPlayltem 
refers to a plurality of clips, as shown in Fig. 9 or 10 . 
[0075] 

The num_of^Clip_entries indicates the number of clips, 
and designates clips other than those having 

Clip__Information_f ile_name [0] [subClip_entiry_id] . That is, 
num_of _Clip_entries designates clips, such as those having 
Clip_Inf ormation_f ile_name [1] , Clip_Inf ormation_f ile__name [2] , 
and so on, other than those having 

Clip_Information_f ile_name [0] . The SubPlayltem also 
includes Clip_codec_ident if ier [subClip__entry_id] for 
specifying the codec method for the clip, 
ref_to_STC_id [subClip_entry_id] , which is information 
concerning the STC non- sequential points (non- sequential 
points of the system time base), and reserved_f or_f uture_use . 
[0076] 

The SubPlayItem__IN_time, SubPlayItem_OUT_time , 
sync_PlayItem_id, and sync_start_PTS_of _PlayItem are used in 
common for the clips referred to by SubPlayltem ( ) . In the 
example in Fig. 9, SubPlayItem_IN_time, SubPlayItem__OUT_time , 
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sync_PlayItem_id, and sync_start_PTS_of _PlayItem are used in 
common for SubClip_entry_id=0 through N. The Text based 
subtitle for the selected SubClip_entry_id is played back 
based on the SubPlayItem_IN_time, SubPlayItem_OUT_time, 
sync_PlayItem_id, and sync_start_PTS_of_PlayItem. 
[0077] 

The numeric value is sequentially assigned to 
SubClip_entry_id from 1 in the order in which 
Clip_Inf ormation_f ile_name [subClip_entry_id] in the 
SubPlayltem appears. The subClip_entry_id of 
Clip_Inf ormation_f ile_name [0] is 0 . 
[0078] 

Fig. 14 illustrates the syntax of PlayltemO . 
[0079] 

The "length" is a 16 -bit unsigned integer indicating 
the number of bytes from immediately after the length field 
to the end of PlayltemO. Clip_Inf ormation_f ile_name [0] is 
a field for specifying the clip referred to by PlayltemO . 
In the example in Fig. 7, the main clip AV stream file is 
referred to by Clip_Inf ormation_f ile_name [0] . The 
PlayltemO also includes Clip_codec_identif ier [0] specifying 
the codec method for the clip, reserved_f or_f uture_use , 
is_multi_angle , connect ion_condit ion, and ref _to_STC_id [0] , 
which is information concerning the STC non- sequential 
points (non- sequential points of the system time base) . The 
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PlayltemO also includes IN_time and OUT_tiTne for specifying 
the playback zone of the play item in the clip. In the 
example in Fig. 7, IN_time and OUT_time specify the playback 
zone of the main clip AV stream file. The PlayltemO also 
includes UO_mask_table ( ) , PlayItem_random_access_mode , and 
still_mode. A description of a case where is_multi_angle 
indicates a plurality of angles is not given here, since 
such a case is not directly related to the present invention. 
[0080] 

The STN_table{) in PlayltemO provides a mechanism for 
allowing a user, if the target Playltem and at least one Sub 
Path to be played back in association with the target 
Playltem are provided, to select from the streams contained 
in the clip referred to by the Playltem and the clips 
referred to by at least one Sub Path when the user switches 
sound or subtitles. 
[0081] 

Fig. 15 illustrates the syntax of STN_table(). The 
STN_table() is set as an attribute of Playltem. 
[0082] 

The "length" is a 16 -bit unsigned integer indicating 
the number of bytes from immediately after the length field 
to the end of STN_table () . After the "length", 16-bit 
reserved_f or_f uture_use follows . The 

number of video stream entries indicates the number of 
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streams provided with video_streaTn_id entered (registered) 
in STN_table(). The video_stream__id is information for 
identifying the video streams. The video_stream_number is 
the video stream number that can be seen by the user when 
switching video. The number_of _audio_stream_entries 
indicates the number of streams provided with 
audio_stream_id entered (registered) in STN_table() . The 
audio_stream_id is information for identifying the audio 
streams. The audio_stream_number is the audio stream number 
that can be seen by the user when switching sound. 
[0083] 

The number_of_PG_txtST_stream_entries indicates the 
number of streams provided with PG_txtST_stream_id entered 
in STN_table(). In STN_table() shown in Fig. 15, streams 
(PG, Presentation Graphics streams) in which bitmap 
subtitles, such as DVD sub-pictures, are run- length coded, 
and text subtitle files (txtST) are entered. The 
PG_txtST__stream_id is information for identifying the 
subtitle streams, and PG_txtST_stream_number is the subtitle 
stream number (text subtitle stream number) that can be seen 
by the user when switching subtitles. 
[0084] 

The num_of_IG_stream_entries indicates the number of 
streams provided with IG_stream_id entered in STN_table(). 
In STN_table() shown in Fig. 15, interactive graphics 
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streams are entered. The IG_stream_id is information for 
identifying the interactive graphics streams. The 
IG_stream_number is the graphics stream number that can be 
seen when switching graphics. 
[0085] 

The syntax of stream__entry ( ) is discussed below with 
reference to Fig. 16. 
[0086] 

The "type" is an 8-bit field indicating the type of 
information required for uniquely specifying the stream 
provided with the above-described stream number . 
[0087] 

If type=l, a packet ID (PID) is designated for 
specifying one elementary stream from a plurality of 
elementary streams multiplexed into the clip (Main Clip) 
referred to by the Playltem. The 

ref_to_stream_PID_of_mainClip indicates this PID. That is, 
if type=l, the stream can be determined only by specifying 
the PID in the main clip AV stream file. 
[0088] 

If type=2, when the SubPath refers to one clip in which 
only one elementary stream is multiplexed, the SubPath_id of 
the Sub Path is designated for specifying that elementary 
stream. The ref_to_SubPath_id indicates the Sub Path_id. 
Type=2 is used when only one audio stream is referred to by 
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the Sub Path, as shown in Fig. 8, i.e., when the SubPlay 

Item contains only one clip. 

[0089] 

If type=3, when SubPath refers to a plurality of clips 
at the same time and only one elementary stream is 
multiplexed in each clip, the SubPath_id and Clip id of the 
SubPath are designated for specifying the elementary stream 
of one clip (Sub Clip) referred to by the SubPath. The 
ref_to_SubPath_id indicates this SubPath_id, and 
ref_to__subClip_entry_id indicates this Clip id. Type=3 is 
used when one Sub Path refers to a plurality of clips (Text 
based Subtitles), as shown in Fig. 9, i.e., when SubPlayltem 
contains a plurality of clips. 
[0090] 

If type=4, when SubPath refers to a plurality of clips 
at the same time and a plurality of elementary streams are 
multiplexed in each clip, the SubPath_id, Clip id, and 
packet ID (PID) of the SubPath are designated for specifying 
one of the plurality of elementary streams of one clip (Sub 
Clip) referred to by the SubPath. The ref _to_SubPath_id 
indicates this SubPath_id, ref_to_subClip_entry_id indicates 
this Clip id, and ref_to_stream_PID_of _subClip indicates 
this PID. Type=4 is used when a plurality of clips are 
referred to in the SubPlayltem and when a plurality of 
elementary streams are referred to by each clip. 
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[0091] 

When Playltem and at least one SubPath played back in 
association with the Playltem are provided, the use of types 

(type=l through type=4) makes it possible to specify one 
elementary stream from the clip referred to by the Playltem 
and the clips referred to by at least one SubPath. It 
should be noted that type=l indicates the clip (main clip) 
referred to by the Main Path, and type=2 through 4 indicate 
the clip (sub clip) referred to by the Sub Path. In Fig. 16, 
four types are provided for specifying elementary streams. 
However, only two types, and more specifically, the type 

(type=l in Fig. 16) for specifying the elementary stream 
multiplexed into the main clip and the type (types=2 through 
4 in Fig. 16) for specifying the elementary stream of the 
clip used by the Sub Path, may be provided. The syntax of 
stream__entry 0 of such a case is described below with 
reference to Fig. 33. 

[0092] 

In Fig. 33, the "type" is an 8-bit field indicating the 
type of information required for uniquely specifying the 
stream provided with the above -described stream number. 
More specifically, the 8-bit type field is used for 
designating the type of a database for specifying the 
elementary stream referred to by the stream number of 
stream_entry ( ) . In the example shown in Fig. 33, the type 
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is divided into two types, as shown in Fig. 34. In Fig. 34, 
type=l is the type (type=l in Fig. 16) for specifying the 
elementary stream multiplexed into the main clip, and the 
type=2 is the type (types=2 through 4 in Fig. 16) for 
specifying the elementary stream of the clip used by the Sub 
Path. 
[0093] 

Type=l in Fig. 33 is used for specifying an elementary 
stream of the clip (main clip) used by the Playltem. More 
specifically, when type=l, the packet ID (PID) is designated 
for specifying one of a plurality of elementary streams 
multiplexed into the clip (main clip) referred to by the 
Playltem. The ref__to_stream_PID_of__mainClip indicates this 
PID. In other words, when type=l, the stream can be 
determined only by specifying the PID in the main clip AV 
stream file. 
[0094] 

Type=2 in Fig. 33 is used for specifying the elementary 
stream of the clip used by the SubPath accompanying the 
Playltem. In the case of type=2, for example, when the 
SubPath refers to one clip in which only one elementary 
stream is multiplexed (type=2 in Fig. 16) , or when the 
SubPath refers to a plurality of clips at the same time and 
only one elementary stream is multiplexed in each clip 
(type=3 in Fig. 16) , or when the SubPath refers to a 
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plurality of clips at the same time and when a plurality of 
elementary streams are multiplexed in each clip (type=4 in 
Fig. 16) , the SubPath_id, Clip id, and packet ID (PID) are 
designated for specifying the elementary stream. 
[0095] 

Although in Fig. 33, when type=2, three IDs, such as 
SubPath_id, Clip id, and packet ID (PID) , are specified, it 
is not necessary to specify all the three IDs. For example, 
when the SubPath refers to one clip in which only one 
elementary stream is multiplexed (type=2 in Fig. 16) , it is 
sufficient if the SubPath_id of the SubPath is designated 
for specifying the elementary stream. When the SubPath 
refers to a plurality of clips at the same time and when 
only one elementary stream is multiplexed in each clip 
(type=3 in Fig. 16) , it is sufficient if the SubPath_id and 
Clip id of the SubPath are designated for specifying the 
elementary stream of the clip (Sub Clip) referred to by the 
SubPath. When the SubPath refers to a plurality of clips at 
the same time and when a plurality of elementary streams are 
multiplexed in each clip (type=4 in Fig. 16), it is 
necessary that the SubPath__id, Clip id, and packet ID (PID) 
of the SubPath are designated for specifying one of the 
plurality of elementary streams of one clip (Sub Clip) 
referred to by the SubPath. That is, when type=2 in Fig. 33 
or 34, among the SubPath_id, Clip id, and packet ID (PID), 
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it is sufficient if at least Subpath_id is designated. 
[0096] 

When the Playltem and at least one SubPath played back 
in association with the Playltem are provided, the use of 
types (types=l and 2) makes it possible, as shown in Figs. 
33 and 34, to specify one elementary stream from the clip 
referred to by the Playltem and clips referred to by at 
least one SubPath. 
[0097] 

Referring back to a description of STN_table() in Fig. 
15, in the FOR loop of the video stream ID (video_stream_id) , 
video_stream_id is assigned from 0 to a video elementary 
stream specified for each stream_entry () . Instead of the 
video stream ID (video_stream_id) , the video stream number 

(video_stream_number) may be used, in which case, the 
video_stream_number is assigned from 1. That is, the number 
obtained by adding one to video_stream_id is the 
video_stream_number . The video stream number is assigned 
from 1 since video_stream_number is the video stream number 
that can be seen by the user when switching video. 

[0098] 

Similarly, in the FOR loop of the audio stream ID 
(audio_stream_id) , audio_stream_id is assigned from 0 to an 
audio elementary stream specified for each stream_entry ( ) . 
As in the video stream, instead of the audio stream ID 
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(audio_stream_id) , the audio stream number 
(audio_stream__number) may be used, in which case, the 
audio_stream_number is assigned from 1. That is, the number 
obtained by adding one to audio_stream_id is the 
audio_stream_number . The audio stream number is assigned 
from 1 since audio_stream_number is the audio stream number 
that can be seen by the user when switching sound. 
[0099] 

Similarly, in the FOR loop of the subtitle stream ID 
(PG_txtST_stream_id) , PG_txtST_stream__id is assigned from 0 
to a bitmap subtitle or text subtitle elementary stream 
specified for each stream_entry ( ) . As in the video stream, 
instead of the subtitle stream ID (PG_txtST_stream_id) , the 
subtitle stream number (PG_txtST_stream_number) may be used, 
in which case, the PG_txtST_stream_number is assigned from 1. 
That is, the number obtained by adding one to 
PG_txtST_stream_id is the PG__txtST__stream_number . The 
subtitle stream number is assigned from 1 since 
PG_txtST_stream_number is the text subtitle stream number 
that can be seen by the user when switching subtitles. 
[0100] 

Similarly, in the FOR loop of the graphics stream ID 
(IG_stream__id) , IG_stream_id is assigned from 0 to an 
interactive graphics elementary stream specified for each 
stream_entry 0 . As in the video stream, instead of the 
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graphics stream ID (IG__stream_id) , the graphics stream 
number (IG_stream_number) may be used, in which case, the 
IG_stream__number is assigned from 1. That is, the number 
obtained by adding one to IG_stream_id is the 
IG_stream_number . The IG_stream_number is assigned from 1 
since IG__stream_number is the graphics stream number that 
can be seen by the user when switching graphics. 
[0101] 

The stream_attribute 0 in the STN_table() shown in Fig. 
15 is as follows - 
[0102] 

The stream__attribute ( ) in the FOR loop of the video 
stream ID (video_stream_id) provides stream, attribute 
information concerning one video elementary stream specified 
by each stream_entry ( ) . That is, in stream_attribute ( ) , 
stream attribute information concerning one video elementary 
stream specified by each stream__entry ( ) is indicated. 
[0103] 

Similarly, the stream_attributes ( ) in the FOR loop of 
the audio stream ID (audio_stream_id) provides stream 
attribute information concerning at least one audio 
elementary stream specified by each stream__entry ( ) . That is, 
in stream_attribute ( ) , stream attribute information 
concerning at least one audio elementary stream specified by 
each stream_entry 0 is indicated. 
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[0104] 

Similarly, the stream_attribute ( ) in the FOR loop of 
the subtitle stream ID (PG_txtST_stream_id) provides stream 
attribute information concerning one bitmap subtitle 
elementary stream or one text subtitle elementary stream 
specified by each stream_entry ( ) . That is, in 
stream_attribute 0 , stream attribute information concerning 
one bitmap subtitle elementary stream specified by each 
stream_entry ( ) is indicated. 
[0105] 

Similarly, the stream_attribute ( ) in the FOR loop of 
the graphics stream ID (IG_stream_id) provides stream 
attribute information concerning one interactive graphics 
elementary stream specified by each stream__entry ( ) . That is, 
in stream_attribute 0 , stream attribute information 
concerning one interactive graphics elementary stream 
specified by each stream_entry ( ) is indicated. 
[0106] 

The syntax of stream__attribute ( ) is discussed below 
with reference to Fig. 17. 
[0107] 

The "length" is a 16 -bit unsigned integer indicating 
the number of bytes from the length field to the end of 
stream__attribute 0 . 
[0108] 
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The stream_coding_type indicates the coding type of 
elementary stream, as shown in Fig, 18. The coding types of 
elementary streams include MPEG-2 video stream, HDMV LPCM 
audio, Dolby AC-3 audio, dts audio. Presentation graphics 
stream. Interactive graphics stream, and Text subtitle 
stream. 
[0109] 

The video_format indicates the video format of a video 
elementary stream, as shown in Fig. 19. The video formats 
of the video elementary streams include 480i, 576i, 48 Op, 
10801, 720p, and 1080p. 
[0110] 

The frame_rate indicates the frame rate of a video 
elementary stream, as shown in Fig. 20. The frame rates of 
the video elementary streams include 24000/1001, 24, 25, 
30000/1001, 50, and 60000/1001. 
[0111] 

The aspect^ratio indicates the aspect ratio of a video 
elementary stream, as shown in Fig. 21. The aspect ratios 
of the video elementary streams include 4:3 display aspect 
ratio and 16:9 display aspect ratio. 
[0112] 

The audio_presentation_type indicates the presentation 
type of an audio elementary stream, as shown in Fig. 22. 
The presentation types of audio elementary streams include 
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single mono channel, dual mono channel, stereo (2 -channel ) , 
and mul t i - channe 1 . 
[0113] 

The sampling_f requency indicates the sampling frequency 
of an audio elementary stream, as shown in Fig. 23. The 
sampling frequencies of the audio elementary streams include 
48 kHz and 96 kHz. 
[0114] 

The audio_language_code indicates the language code 
(for example, Japanese, Korean, or Chinese) of an audio 
elementary stream , 
[0115] 

The PG_language_code indicates the language code (for 
example, Japanese, Korean, or Chinese) of a bitmap subtitle 
elementary stream . 
[0116] 

The IG_language_code indicates the language code (for 
example, Japanese, Korean, or Chinese) of an interactive 
graphics elementary stream. 
[0117] 

The textST_language_code indicates the language code 
(for example, Japanese, Korean, or Chinese) of a text 
subtitle elementary stream. 
[0118] 

The character code indicates the character code of a 
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text subtitle elementary stream, as shown in Fig. 24. The 
character codes of text subtitle elementary streams include 
Unicode VI . 1 (ISO 10646-1), Shift JIS (Japanese), KSC 5601- 
1987 including KSC 5653 for Roman character (Korean) , 
GB18030-2000 (Chinese) , GB2312 (Chinese) , and BIGS (Chinese) . 
[0119] 

The syntax of stream__attribute ( ) shown in Fig. 17 is 
specifically described below with reference to Figs. 17, and 
18 through 24. 
[0120] 

If the coding type (stream_coding_type in Fig. 17) of 
the elementary stream is the MPEG-2 video stream (Fig. 18) , 
stream_attribute 0 includes the video format (Fig. 19), the 
frame rate (Fig. 20), and the aspect ratio (Fig. 21) of the 
elementary stream. 
[0121] 

If the coding type (stream_coding_type in Fig. 17) of 
the elementary stream is the HDMV LPCM audio, Dolby AC- 3 
audio, or dts audio (Fig. 18), stream_attribute ( ) includes 
the audio presentation type (Fig. 22), the sampling 
frequency (Fig. 23) , and the language code of the audio 
elementary stream. 
[0122] 

If the coding type (stream_coding_type in Fig. 17) of 
the elementary stream is the Presentation graphics stream 
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(Fig. 18), stream_attribute 0 includes the language code of 
the bitmap subtitle elementary stream. 
[0123] 

If the coding type (stream_coding_type in Fig. 17) of 
the elementary stream is the Interactive graphics stream 
(Fig. 18), stream_attribute ( ) includes the language code of 
the interactive graphics elementary stream. 
[0124] 

If the coding type (stream_coding_type in Fig. 17) of 
the elementary stream is the Text subtitle stream (Fig. 18) , 
stream__attribute ( ) includes the character code (Fig. 24) and 
the language code of the text subtitle elementary stream. 
[0125] 

The attribute information is not restricted to the 
above -described types. 
[0126] 

In this manner, if Playltem and at least one Sub Path 
played back in association with the Playltem are provided, 
by referring to the clip referred to by the Playltem and 
clips referred to by at least one Sub Path, attribute 
information concerning an elementary stream specified by 
stream_entry ( ) can be defined by stream_attribute ( ) . 
[0127] 

By checking the attribute information 
(stream_attribute ( ). ) , the playback apparatus can determine 
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whether it has a function of playing back the corresponding 
elementary stream. Also, by checking the attribute 
information, the playback apparatus can select the 
elementary streams in accordance with the initial 
information concerning the language set in the playback 
apparatus . 
[0128] 

It is now assumed, for example, that the playback 
apparatus has a function of playing back bitmap subtitle 
elementary streams without a function of playing back text 
subtitle elementary streams. In this case, in response to 
an instruction to switch the languages from the user, the 
playback apparatus sequentially selects only bitmap subtitle' 
elementary streams from the FOR loop of the subtitle stream 
ID (PG_txtST_stream_id) and plays back the selected 
elementary streams . 
[0129] 

If the initial information concerning the language set 
in the playback apparatus is Japanese, in response to an 
instruction to switch sound from the user, the playback 
apparatus sequentially selects only audio elementary stream 
whose language code is Japanese from the FOR loop of the 
audio stream ID (Audio_stream_id) and plays back the 
selected elementary streams. 
[0130] 
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As described above, by the provision of STN_table() in 
PlayltemO , if Playltem and at least one Sub Path played 
back in association with the Playltem are provided, the user 
can select a stream to be played back from the clip referred 
to by the Playltem and clips referred to by at least one Sub 
Path when switching sound or subtitles. Thus, interactive 
operations can be performed for streams or data files 
different from a main AV stream to be played back. 
[0131] 

Since one PlayList includes a plurality of Sub Paths 
and each Sub Path refers to a SubPlayltem, AV streams having 
high extensibility and high flexibility are implemented. 
That is, SubPlayltems can be added afterwards. For example, 
if Playlist including a clip AV stream file referred to by 
the Main Path is replaced by PlayList including the clip AV 
stream file and a new Sub Path, the user can refer to, based 
on the new playlist, not only the clip AV stream file 
referred to by the Main Path, but also clip AV stream files 
different from the clip AV stream file referred to by the 
Main Path. Thus, the AV streams have high extensibility. 
[0132] 

A playback apparatus to which the present invention is 
applied is described below. Fig. 25 is a block diagram 
illustrating an example of the configuration of the playback 
apparatus 20 to which the present invention is applied. The 
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playback apparatus 20 is the playback apparatus 2 0 for 
playing back Playlist including the above -described Main 
Path and Sub Path. 
[0133] 

The playback apparatus 2 0 includes a storage drive 31, 
a switch 32, an AV decoder 33, and a controller 34. 
[0134] 

In the example shown in Fig. 25, the controller 34 
first reads out a PlayList file from the storage drive 31, 
and reads an AV stream or AV data from a recording medium, 
such as an HDD, Blu-ray disc, or DVD, via the storage drive 
31 on the basis of the information concerning the PlayList 
file. The user can give an instruction to switch sound or 
subtitles to the controller 34 by using a user interface. 
The controller 34 reads initial information concerning the 
language set in the playback apparatus 2 0 from a storage 
unit (not shown) . 
[0135] 

The PlayList file includes, not only information 
concerning Main Path and Sub Path, but also STN_table() . 
The controller 34 reads a main clip AV stream file 
(hereinafter referred to as a "main clip") referred to by 
Playltem contained in the PlayList file, a sub clip AV 
stream file (hereinafter referred to as a "sub clip") 
referred to by SubPlayltem, and text subtitle data referred 
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to by SubPlayltem from a recording medium via the storage 
drive 31. The controller 34 controls the playback apparatus 
2 0 to select and play back elementary streams in accordance 
with the playback function of the playback apparatus 2 0 or 
to select and play back elementary streams in accordance 
with the initial information concerning the language set in 
the playback apparatus 20. 
[0136] 

The AV decoder 33 includes buffers 51 through 54, PID 
filters 55 and 56, switches 57 through 59, a background 
decoder 71, an MPEG (Moving Picture Experts Group) 2 video 
decoder 72, a presentation graphics decoder 73, an 
interactive graphics decoder 74, an audio decoder 75, a 
Text-ST composition 76, a switch 77, a background plane 
generator 91, a video plane generator 92, a presentation 
graphics plane generator 93, an interactive graphics plane 
generator 94, a buffer 95, a video data processor 96, and an 
audio data processor 97, 
[0137] 

File data read by the controller 34 is demodulated by a 
demodulator, and the demodulated multiplexed streams are 
then subjected to error correction by an ECC decoder. The 
switch 32 then divides the multiplexed streams subjected to 
error correction according to the stream types and supplies 
the divided streams to the corresponding buffers 51 through 



- 56 - 

S05P0181 

54 under the control of the controller 34 . More 
specifically, under the control of the controller 34, the 
switch 32 supplies background image data to the buffer 51, 
main clip data to the buffer 52, sub clip data to the buffer 
53, and text-ST data to the buffer 54. Then, the buffers 51 
through 54 buffer the background image data, main clip data, 
sub clip data, and text-ST data, respectively, therein. 
[0138] 

The main clip is a stream (for example, a transport 
stream) in which at least one stream of video, audio, bitmap 
subtitle (presentation graphics stream) , and interactive 
graphics streams is multiplexed together with a video stream. 
The sub clip is a stream in which at least one stream of 
audio, bitmap subtitle (presentation graphics stream) , and 
interactive graphics streams is multiplexed. The data of 
the text subtitle data file (Text-ST) may be a multiplexed 
stream, such as a transport stream, but this is not 
essential . 
[0139] 

When reading the main clip, the sub clip, and the text 
subtitle data from the storage drive 31 (recording medium) , 
they can be alternately read in a time-division manner . 
Alternatively, the sub clip or the text subtitle data may be 
entirely preloaded to the buffer (buffer 53 or 54) , 
respectively, before reading the main clip. 
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[0140] 

The playback apparatus 2 0 reads those file data from a 
recording medium via the storage drive 31 to play back video, 
bitmap subtitle, interactive graphics, and audio. 
[0141] 

More specifically, stream data read out from the buffer 
52, which serves as the main clip read buffer, is output to 
the PID (packet ID) filter 55, which is disposed subsequent 
to the buffer 52, at a predetermined time. The PID filter 
55 allocates streams contained in the main clip to the 
corresponding elementary stream decoders, which are disposed 
subsequent to the PID filter 55, according to the PIDs 

(packet IDs) . More specifically, the PID filter 55 supplies 
video streams to the MPEG2 video decoder 72, presentation 
graphics streams to the switch 57, which supplies the 
graphics streams to the presentation graphics decoder 73, 
interactive graphics streams to the switch 58, which 
supplies the graphics streams to the interactive graphics 
decoder 74, and audio streams to the switch 59, which 
supplies the audio streams to the audio decoder 75. 

[0142] 

The presentation graphics streams are, for example, 
bitmap subtitle data, and the text subtitle data is, for 
example, text subtitle data. 
[0143] 
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Stream data read out from the buffer 53, which serves 
as the sub clip read buffer, is output to the PID (packet 
ID) filter 56, which is disposed subsequent to the buffer 53, 
at a predetermined time. The PID filter 56 allocates 
streams contained in the sub clip to the corresponding 
elementary stream decoders, which are disposed subsequent to 
the PID filter 56, according to the PIDs (packet IDs) . More 
specifically, the PID filter 56 supplies presentation 
graphics streams to the switch 57, which supplies the 
graphics streams to the presentation graphics decoder 73, 
interactive graphics streams to the switch 58, which 
supplies the graphics streams to the interactive graphics 
decoder 74, and audio streams to the switch 59, which 
supplies the audio streams to the audio decoder 75, 
[0144] 

Data read out from the buffer 51, which serves as the 
background image data buffer, is supplied to the background 
decoder 71 at a predetermined time. The background decoder 
71 decodes the background image data, and then supplies the 
decoded data to the background plane generator 91. 
[0145] 

The video streams allocated by the PID filter 55 are 
supplied to the video decoder 72, which is disposed 
subsequent to the PID filter 55. The video decoder 72 
decodes the video streams and supplies the decoded video 
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streams to the video plane generator 92 . 
[0146] 

The switch 57 selects one of the presentation graphics 
streams contained in the main clip supplied from the PID 
filter 55 and the presentation graphics streams contained in 
the sub clip supplied from the PID filter 56, and supplies 
the selected presentation graphics streams to the 
presentation graphics decoder 73, which is disposed 
subsequent to the switch 57. The presentation graphics 
decoder 73 decodes the presentation graphics streams and 
supplies them to the switch 77, which further supplies them 
to the presentation graphics plane generator 93 - 
[0147] 

The switch 58 selects one of the interactive graphics 
streams contained in the main clip supplied from the PID 
filter 55 and the interactive graphics streams contained in 
the sub clip, and supplies the selected interactive graphics 
streams to the interactive graphics stream decoder 74, which 
is disposed subsequent to the switch 58. That is, the 
interactive graphics streams simultaneously input into the 
interactive graphics decoder 74 are streams separated from 
the main clip or the sub clip. The interactive graphics 
decoder 74 decodes the interactive graphics streams, and 
supplies the decoded streams to the interactive graphics 
plane generator 94 . 
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[0148] 

The switch 59 selects one of the audio streams 
contained in the main clip supplied from the PID filter 55 
and the audio streams contained in the sub clip and supplies 
the selected audio streams to the audio decoder 75, which is 
disposed subsequent to the switch 59. That is, the audio 
streams simultaneously input into the audio decoder 75 are 
streams separated from the main clip or the sub clip. The 
audio decoder 7 5 decodes the audio stream and supplies the 
decoded audio stream to the audio data processor 97 . 
[0149] 

Sound data selected by the switch 32 is supplied to the 
buffer 95 and is buffered therein. The buffer 95 supplies 
the sound data to the audio data processor 97 at a 
predetermined time. The sound data is, for example, effect 
sound that can be selected from a menu. 
[0150] 

Data read from the buffer 54, which serves as the text 
subtitle read buffer, is output to the text subtitle 
composition (decoder) 76, which is disposed subsequent to 
the buffer 54, at a predetermined time. The text subtitle 
composition 76 decodes the text-ST data and supplies the 
decoded data to the switch 77. 
[0151] 

The switch 77 selects one of the presentation graphics 
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streams decoded by the presentation graphics decoder 73 and 
the Text-ST (text subtitle data) , and supplies the selected 
data to the presentation graphics plane generator 93. That 
is, subtitle images simultaneously supplied to the 
presentation graphics plane generator 93 are those output 
from the presentation graphics decoder 73 or from the text 
subtitle (Text-ST) composition 76. Presentation graphics 
streams simultaneously input into the presentation graphics 
decoder 73 are streams separated from the main clip or from 
the sub clip (selected by the switch 57) . Accordingly, the 
subtitle images simultaneously input into the presentation 
graphics plane generator 93 are presentation graphics 
streams from the main clip, presentation graphics streams 
from a sub clip or text subtitle data. 
[0152] 

The background plane generator 91 generates a 
background plane, which serves as, for example, a wallpaper 
image when a video image is displayed by reducing the size 
thereof, on the basis of the background image data supplied 
from the background decoder 71, and supplies the generated 
background plane to the video data processor 96. The video 
plane generator 92 generates a video plane based on the 
video data supplied from the MPEG2 video decoder 72, and 
supplies the generated video plane to the video data 
processor 96. The presentation graphics plane generator 93 
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generates a presentation graphics plane, which serves as, 
for example, a rendering image, on the basis of the data 

(presentation graphics streams or text subtitle data) 
selected by the switch 77, and supplies the generated 
presentation graphics plane to the video data processor 96. 
The interactive graphics plane generator 94 generates an 
interactive graphics plane based on the interactive graphics 
stream data supplied from the interactive graphics decoder 
74, and supplies the generated interactive graphics plane to 
the video data processor 96 . 

[0153] 

The video data processor 96 combines the background 
plane from the background plane generator 91, the video 
plane from the video plane generator 92, the presentation 
graphics plane from the presentation graphics plane 
generator 93, and the interactive graphics plane from the 
interactive graphics plane generator 94, and outputs the 
combined plane as a video signal. The audio data processor 
97 combines the audio data from the audio decoder 75 with 
the sound data from the buffer 95, and outputs the combined 
data as an audio signal . 
[0154] 

The switches 57 through 59 and the switch 77 select 
data according to the selection by the user via the user 
interface or depending on the type of file containing target 
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data. For example, if audio streams are contained only in 
sub-clip AV stream files, the switch 59 changes the 
selection to the sub clip side. 
[0155] 

The playback processing performed by the playback 
apparatus 20 shown in Fig. 25 is described below with 
reference to the flowcharts in Figs. 26 through 28. This 
processing is started when an instruction to play back a 
predetermined AV stream is given by a user via a user 
interface . 
[0156] 

In step Sll, the controller 34 reads a PlayList file 
recorded on a recording medium or a HDD (Hard Disk Drive) 

(not shown) via the storage drive 31. For example, the 
PlayList file discussed with reference to Fig. 11 is read. 

[0157] 

In step S12, the controller 34 reads a main clip, a sub 
clip, and text subtitle data (text-ST data) . More 
specifically, the controller 34 reads the corresponding main 
clip based on the Playltem contained in the PlayList 
discussed with reference to Fig. 11. The controller 34 also 
reads a sub clip and text subtitle data based on the 
SubPlayltem discussed with reference to Figs. 12 and 13, 
which is referred to by the SubPath contained in the 
PlayList . 
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[0158] 

In step S13, the controller 34 controls the switch 32 
to supply the read data (main clip, sub clip, and text 
subtitle data) to the corresponding buffers 51 through 54. 
More specifically, the controller 34 controls the switch 32 
to supply the background image data to the buffer 51, the 
main clip data to the buffer 52, the sub clip data to the 
buffer 53, and the Text-ST data to the buffer 54, 
[0159] 

In step S14, the switch 32 is changed under the control 
of the controller 34. Then, the bacJcground data is supplied 
to the buffer 51, the main clip data is supplied to the 
buffer 52, the sub clip data is supplied to the buffer 53, 
and the text subtitle data is supplied to the buffer 54. 
[0160] 

In step S15, the buffers 51 through 54 buffer the 
supplied data therein. More specifically, the buffer 51 
buffers the bacJcground image data, the buffer 52 buffers the 
main clip data, the buffer 53 buffers the sub clip data, and 
the buffer 54 buffers the Text-ST data. 
[0161] 

In step S16, the buffer 51 outputs the baclcground image 
data to the background decoder 71. 
[0162] 

In step S17, the buffer 52 outputs the main clip stream 
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data to the PID filter 55. 
[0163] 

In step S18, the PID filter 55 allocates the elementary 
streams to the corresponding elementary stream decoders 
based on the PIDs attached to the TS packets forming the 
main clip AV stream file. More specifically, the PID filter 
55 supplies video streams to the MPEG2 video decoder 72, 
presentation graphics streams to the switch 57, which 
supplies the streams to the presentation graphics decoder 73, 
interactive graphics streams to the switch 58, which 
supplies the streams to the interactive graphics decoder 74, 
and audio streams to the switch 59, which supplies the 
streams to the audio decoder 75. That is, the video streams, 
presentation graphics streams, interactive graphics streams, 
and audio streams are provided with different PIDs. 
[0164] 

In step S19, the buffer 53 outputs the sub clip stream 
data to the PID filter 56. 
[0165] 

In step S20, the PID filter 56 allocates the elementary 
streams to the corresponding decoders based on the PIDs. 
More specifically, the PID filter 56 supplies presentation 
graphics streams to the switch 57, which supplies the 
streams to the presentation graphics decoder 73, interactive 
graphics streams to the switch 58, which supplies the 
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streams to the interactive graphics decoder 74, and audio 
streams to the switch 59, which supplies the streams to the 
audio decoder 75. 
[0166] 

In step S21, the switches 57 through 59, which are 
disposed subsequent to the PID filters 55 and 56, select one 
of the main clip and the sub clip under the control of the 
controller 34 via the user interface. More specifically, 
the switch 57 selects presentation graphics streams of the 
main clip or those of the sub clip supplied from the PID 
filter 55, and supplies the selected streams to the 
presentation graphics decoder 73, which is disposed 
subsequent to the switch 57. The switch 58 selects 
interactive graphics streams of the main clip or those of 
the sub clip supplied from the PID filter 55, and supplies 
the selected streams to the interactive graphics decoder 74, 
which is disposed subsequent to the switch 58. The switch 
59 selects audio streams of the main clip or those of the 
sub clip supplied from the PID filter 55, and supplies the 
selected streams to the audio decoder 75, which is disposed 
subsequent to the switch 59- 
[0167] 

In step S22, the buffer 54 outputs text subtitle data 
to the text subtitle composition 76. 
[0168] 
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In step S23, the background decoder 71 decodes 
background image data and supplies the decoded data to the 
background plane generator 91. 
[0169] 

In step S2A, the MPEG2 video decoder 72 decodes the 
video streams and supplies the decoded streams to the video 
plane generator 92 . 
[0170] 

In step S2S, the presentation graphics decoder 73 
decodes the presentation graphics streams selected by the 
switch 57, and outputs the decoded streams to the switch 77, 
which is disposed subsequent to the presentation graphics 
decoder 73 • 
[0171] 

In step S2 6, the interactive graphics decoder 74 
decodes the supplied interactive graphics streams selected 
by the switch 58, and outputs the decoded streams to the 
interactive graphics plane generator 94, which is disposed 
subsequent to the interactive graphics decoder 74 . 
[0172] 

In step S2 7, the audio decoder 75 decodes the supplied 
audio data selected by the switch 59 and outputs the decoded 
data to the audio data processor 97, which is disposed 
subsequent to the audio decoder 75. 
[0173] 
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In step S28, the Text-ST composition 76 decodes text 
subtitle data and outputs the decoded data to the switch 77, 
which is disposed subsequent to the Text-ST composition 76. 
[0174] 

In step 329, the switch 77 selects data from the 
presentation graphics decoder 73 or the Text-ST composition 
76- More specifically, the switch 77 selects the 
presentation graphics streams decoded by the presentation 
graphics decoder 73 or the Test-ST (text subtitle data) from 
the Text-ST composition 76, and supplies the selected data 
to the presentation graphics plane generator 93, 
[0175] 

In step S30, the background plane generator 91 
generates a baclcground plane based on the bacJcground image 
data supplied from the baclcground decoder 71. 
[0176] 

In step S31, the video plane generator 92 generates a 
video plane based on the video data supplied from the MPEG2 
video decoder 72 . 
[0177] 

In step S32, the presentation graphics plane generator 
93 generates a presentation graphics plane based on the data 
selected by the switch 77 and supplied from the presentation 
graphics decoder 73 or the Text-ST composition 76 in step 
S29. 
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[0178] 

In step S33, the interactive graphics plane generator 
94 generates an interactive graphics plane based on the 
interactive graphics stream data supplied from the 
interactive graphics decoder 74 . 
[0179] 

In step S34, the buffer 95 buffers the sound data 
selected and supplied in step S14 and supplies it to the 
audio data processor 97 at a predetermined time. 
[0180] 

In step S35, the video data processor 97 combines the 
planes and outputs the combined data. More specifically, 
the video data processor 97 combines the data from the 
background plane generator 91, the video plane generator 92, 
the presentation graphics plane generator 93, and the 
interactive graphics plane generator 94, and outputs the 
combined data as video data . 
[0181] 

In step S3 6, the audio data processor 97 combines the 
audio data with the sound data, and outputs the resulting 
data. 
[0182] 

According to the processing shown in Figs. 26 through 
28, by referring to the main clip referred to by the main 
path included in the PlayList, a sub clip referred to by the 
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corresponding Sub Path included in the PlayList, and the 
text subtitle data, the corresponding data is played back. 
By providing the Main Path and the Sub Path in the PlayList, 
a clip AV stream file, which is different from the main clip 
AV stream file specified by the Main Path, can be specified 
by the Sub Path. Thus, sub clip data, which is different 
from the main clip specified by the Playltem of the Main 
Path, can be played back together with (in synchronization 
with) the main clip data contained in the main clip. 
[0183] 

In Figs. 26 through 28, the order of steps S16 and S17 
may be reversed or steps S16 and SI 7 may be executed in 
parallel. Similarly, the order of steps S18 and S20 may be 
reversed or steps S18 and S20 may be executed in parallel. 
The order of steps S2 3 through S2 8 may be reversed or steps 
S23 through S28 may be executed in parallel. The order of 
steps S3 0 through S3 3 may be reversed or steps S3 0 through 
S33 may be executed in parallel. The order of steps S35 and 
S3 6 may be reversed or steps S3 5 and S3 6 may be executed in 
parallel. That is, in Fig. 25, the elements disposed 
vertically on the same layer, i.e., the processing jobs of 
the buffers 51 through 54, those of the switches 57 through 
59, those of the decoders 71 through 76, those of the plane 
generators 91 through 94, and those of the video data 
processor 96 and the audio data processor 97 may be executed 
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in parallel, and the order thereof is not particularly 
restricted. 
[0184] 

Processing performed by the playback apparatus 2 0 when 
an instruction to switch sound or subtitles is given is now 
described with reference to the flowchart in Figs. 2 9 and 30. 
[0185] 

Reference is first given to the flowchart in Fig. 2 9 to 
discuss processing when an instruction to switch sound is 
given by the user. This processing is executed when the 
playback processing shown in Figs. 26 through 28 is being 
performed. 
[0186] 

In step S51, the controller 34 obtains an order list of 
audio stream numbers (may be IDs) . More specifically, the 
controller 34 refers to STN_table() of PlayltemO discussed 
with reference to Fig. 14 to obtain the order list of audio 
stream numbers (IDs) entered in STN_table() discussed with 
reference to Fig. 15. This processing is executed when the 
playback processing shown in Figs. 26 through 28 is started. 
[0187] 

In response to an instruction to switch sound given by 
the user via the user interface, in step S52 , the controller 
34 receives the instruction to switch sound given by the 
user. That is, in Fig. 29, step S51 has been executed, and 
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in response to an instruction to switch sound from the user, 
step S52 is executed. 
[0188] 

In step S53, the controller 34 obtains the audio stream 
number subsequent to the audio stream number which is 
currently played back. For example, if the audio stream 
(although it is indicated by Text based subtitle in Fig. 9, 
it is read as the audio stream file in this example) having 
SubClip_entry_id=0 shown in Fig. 9 is played back, the audio 
stream number having SubClip_entry_id=l is obtained. 
[0189] 

In step S54, the controller 34 determines whether the 
playback apparatus 2 0 has a function of playing back the 
audio stream associated with the obtained number. More 
specifically, the controller 34 makes this determination 
based on the information indicated in stream_attribute ( ) 
(Fig. 17) . If it is determined in step S54 that the 
function of playing back the audio stream associated with 
the obtained number is not provided, the process proceeds to 
step S55 in which the controller 34 obtains the stream 
number subsequent to the current stream number. That is, if 
the function of playing back the audio stream associated 
with the current stream number is not provided, the current 
stream number is skipped (which is not to be played back) , 
and the subsequent stream number is obtained- Then, after 
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step S55, the process returns to step S54, and the 
subsequent processing is repeated- That is, the processing 
is repeated until an audio stream number that can be played 
back by the playback apparatus 2 0 can be obtained. 
[0190] 

If it is determined in step S54 that the function of 
playing back the audio stream associated with the obtained 
number is provided, the process proceeds to step S56- In 
step 356, the controller 34 checks whether the audio stream 
is contained in the main clip or a sub clip. In the example 
shown in Fig. 9, since the obtained SubCl ip_entry_id=l is 
referred to by the Sub Path, the controller 34 can determine 
that the audio stream associated with the obtained number is 
contained in the sub clip. 
[0191] 

In step S57, the controller 34 specifies a desired 
audio stream. More specifically, the controller 34 
specifies a desired audio stream contained in the main clip 
or the sub clip associated with the obtained number. More 
specifically, type=3 is specified in STN_table() discussed 
with reference to Fig. 16. 
[0192] 

In step S58, the controller 34 instructs the storage 
drive 31 to read the clip (main clip or sub clip) in which 
the desired audio stream is multiplexed. The storage drive 
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31 reads the target clip based on this instruction. 
[0193] 

In step S59, the controller 34 instructs the AV decoder 

33 to play back the audio stream from the read clip. 
[0194] 

In step 360, the AV decoder 33 decodes the audio stream 
and outputs it. More specifically, the audio data decoded 
by the audio decoder 75 and the sound data output from the 
buffer 95 are processed by the audio data processor 97, and 
the resulting data is output as an audio signal . 
[0195] 

According to this processing, the selection performed 
by the switch 59 shown in Fig. 25 in step S21 in Fig. 27 is 
determined. More specifically, if the target clip shown in 
Fig. 29 is the main clip, the switch 59 supplies the audio 
stream supplied from the main side, i.e., the PID filter 55, 
to the audio decoder 75. If the target clip is a sub clip, 
the switch 59 supplies the audio stream supplied from the 
sub side, i.e., the PID filter 56, to the audio decoder 75. 
[0196] 

In this manner, the controller 34 can control switching 
of sound (audio) based on STN_table() of Playltem. By 
referring to stream_attribute of STN_table(), the controller 

34 can control switching of the playback operation by 
selecting streams that can be played back by the playback 
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apparatus 20. 
[0197] 

Although in the processing shown in Fig. 2 9 audio is 
switched based on audio stream numbers, audio may be 
switched based on audio stream IDs {audio_stream_id) . In 
this case, the number obtained by subtracting one from the 
audio stream number is the audio stream ID. 
[0198] 

A description is now given, with reference to the 
flowchart in Fig. 30, of processing when an instruction to 
switch subtitles is given by the user. This processing is 
executed when the playback processing shown in Figs. 26 
through 28 is being performed. 
[0199] 

In step S81, the controller 34 obtains an order list of 
subtitle stream numbers (may be IDs) . More specifically, 
the controller 34 refers to STN_table() of PlayltemO 
discussed with reference to Fig. 14 to obtain the order list 
of subtitle stream ID (PG_txtST_stream_id) entered in 
STN_table 0 discussed with reference to Fig. 15. This 
processing is executed when the playback processing shown in 
Figs. 2 6 through 2 8 is started. 
[0200] 

In response to an instruction to switch subtitles given 
by the user via the user interface, in step S82, the 
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controller 34 receives the instruction to switch subtitles 
given by the user. That is, in Fig. 30, step S81 has been 
executed, and in response to an instruction to switch 
subtitles from the user, step S82 is executed. 
[0201] 

In step S83, the controller 34 obtains the subtitle 
stream number subsequent to the subtitle stream number which 
is currently played back. For example, if the Text based 
subtitle in Fig. 9 having SubClip_entry_id=0 shown in Fig. 9 
is played back, the Text based subtitle number having 
SubClip_entry_id=l is obtained. 
[0202] 

In step S84, the controller 34 determines whether the 
playback apparatus 2 0 has a function of playing back the 
subtitle stream associated with the obtained number. More 
specifically, the controller 34 makes this determination 
based on the information indicated in stream_attribute ( ) 
(Fig, 17) . If it is determined in step S84 that the 
function of playing back the subtitle stream associated with 
the obtained number is not provided, the process proceeds to 
step S85 in which the controller 34 obtains the stream 
number subsequent to the current stream number. That is, if 
the function of playing back the subtitle stream associated 
with the current stream number is not provided, the current 
stream number is skipped (which is not played back) , and the 
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subsequent stream number is obtained. Then, after step S85, 
the process returns to step S84, and the subsequent 
processing is repeated. That is, the processing is repeated 
until a subtitle stream number that can be played back by 
the playback apparatus 2 0 can be obtained. 
[0203] 

If it is determined in step S84 that the function of 
playing back the subtitle stream associated with the 
obtained number is provided, the process proceeds to step 
S86. In step BBS, the controller 34 checks whether the data 
corresponding to the obtained number (subtitle stream number 
subsequent to the currently played back subtitle stream) is 
contained in the main clip (Main Path) , a sub clip (Sub 
Path) , or a text subtitle data file (Sub Path) . 
[0204] 

In step S87, the controller 34 specifies a desired 
presentation graphics stream or text subtitle data. More 
specifically, the controller 34 specifies a desired 
presentation graphics stream contained in the main clip or 
the sub clip or desired text subtitle title data from the 
text subtitle file. 
[0205] 

In step 888, the controller 34 instructs the storage 
drive 31 to read the clip (main clip or sub clip) in which 
the desired presentation graphics stream is multiplexed or 
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to read the desired text subtitle data. 
[0206] 

In step S89, the controller 34 instructs the AV decoder 
33 to play back the presentation graphics stream from the 
read clip or the text subtitle data. 
[0207] 

In step S90, the AV decoder 33 decodes the presentation 
graphics stream or the text subtitle data and outputs the 
subtitle image. More specifically, a plane is generated 
from the decoded presentation graphics stream or text 
subtitle data by the presentation graphics plane generator 
93, and is combined by the video data processor 96 and is 
output as video . 
[0208] 

According to this processing, the selection performed 
by the switch 77 shown in Fig. 25 in step S29 in Fig. 28 is 
determined. More specifically, if the target data in step 
S87 in Fig. 30 is a presentation graphics stream, the switch 
77 supplies the presentation graphics data supplied from the 
presentation graphics decoder 73 to the presentation 
graphics plane generator 93. If the target data is text 
subtitle data, the switch 77 supplies the text subtitle data 
supplied from the Text -ST composition 76 to the presentation 
graphics plane generator 93. The controller 34 can control 
switching of the playback operation by selecting only 
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streams that can be played back by the playback apparatus 20. 
[0209] 

Although in the processing shown in Fig. 30, the 
subtitles are switched based on the subtitle stream numbers, 
the subtitles may be switched based on the subtitle stream 
IDs (PG_txtST_stream_id) . In this case, the number obtained 
by subtracting one from the subtitle stream number is the 
subtitle stream ID. 
[0210] 

By providing the Main Path and the Sub Path in the 
PlayList, audio or subtitles can be selected from streams or 
data files different from the main AV stream when an 
instruction to switch audio or subtitles is given by the 
user. 
[0211] 

The Playltem in the Main Path includes data multiplexed 
into an AV stream file and the Stream Number Definition 
table defining the type of data referred to by the Sub Path 
is provided. Thus, streams having higher extensibility can 
be implemented. 
[0212] 

By referring to stream_attribute in STN_table(), the 
playback apparatus 2 0 can sequentially select and play back 
only streams that can be played back by the playback 
apparatus 20. 
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[0213] 

The above-described processing can be summarized as 
follows . 
[0214] 

The playback apparatus 20 obtains PlayList, which 
serves as playback management information, including Main 
Path, which is the main playback path, indicating the 
position of an AV stream file recorded on a recording medium, 
and Sub Paths, which serve as a plurality of sub playback 
paths, indicating the positions of sub clips including 
accompanying data (for example, audio stream data or bitmap 
subtitle stream file data) played back in synchronization 
with the playback timing of main image data (video stream 
data) included in the AV stream file referred to by the main 
playback path. The playback apparatus 2 0 selects 
accompanying data to be played back, based on an instruction 
from the user, from among accompanying data (for example, 
audio stream file data) played back in synchronization with 
video stream data included in the AV stream file referred to 
by the Main Path and accompanying data (for example, audio 
stream file data) included in the sub clips referred to by 
the Sub Paths. In the processing shown in Fig. 2 9 or 30, an 
instruction to switch subtitles or audio is given. Then, 
the playback apparatus 2 0 determines whether the playback 
apparatus 2 0 has a playback function of playing back the 
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selected accompanying data (for example, audio stream file 
data) . The controller 34 can determine whether the 
controller 34 (playback apparatus 20) can play back the 
accompanying data by referring to stream_attribute in 
STN_table 0 . If it is determined that the playback 
apparatus 2 0 has a function of playing back the selected 
accompanying data and if the accompanying data is contained 
in a sub clip referred to by a Sub Path, the sub clip 
referred to by the Sub Path is read and combined with the 
main AV stream file (main clip) referred to by the Main Path, 
and is played back. For example, if, as the accompanying 
data to be played back, audio stream file data referred to 
by a Sub Path is selected by the user (if an instruction to 
switch audio is given by the user) , the playback apparatus 
2 0 combines the audio stream file data referred to by the 
Sub Path with a main clip AV stream file, i.e., an MPEG2 
video stream file, a presentation graphics stream file, or 
an interactive graphics stream file, and plays back the 
combined data. That is, the decoded audio stream file 
selected by the user is played back as audio. 
[0215] 

As described above, since the PlayList includes the 
Main Path and Sub Path, which refer to different clips, the 
extensibility of streams can be achieved. Since one Sub 
Path can refer to a plurality of files (for example. Figs. 9 
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and 10) , the user can select from among a plurality of 
different streams . 
[0216] 

Further, in the Playltem of the Main Path, STN_table() 
shown in Fig. 15 is disposed as a table defining 
accompanying data multiplexed (included) in the AV stream 
file referred to by the Main Path and accompanying data 
referred to by the Sub Paths. Thus, streams having higher 
extensibility can be implemented. The Sub Path can be 
easily extended by being entered in STN_table() . 
[0217] 

The provision of stream__attribute ( ) shown in Fig. 17, 
which is attribute information concerning streams, in 
STN_table() makes it possible to determine whether the 
selected stream can be played bacl<: by the playback apparatus 
20. Also, by referring to stream_attribute () , only streams 
that can be played back by the playback apparatus 2 0 can be 
selected and played back. 
[0218] 

The Sub path includes SubPath_type indicating the type 
of Sub Path (such as audio or text subtitle) , as shown in 
Fig. 12, Clip_inf ormation_f ile__name shown in Fig. 13 
indicating the name of the sub clip referred to by the Sub 
Path, and SubPlayItem_IN_time and SubPlayItem_OUT_time shown 
in Fig. 13 indicating the IN point and the OUT point. 
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respectively, of the clip referred to by the Sub Path. 
Accordingly, the data referred to by the Sub Path can be 
precisely specif ied. 
[0219] 

The Sub Path also includes sync_PlayItem_id (for 
example, sync_PlayIteTn_id shown in Fig. 7 or 9) , which 
serves as specifying information for specifying the AV 
stream file on the Main Path, for playing back the Sub Path 
simultaneously with the Main Path, and 
sync__start_PTS__of_PlayItem (for example, 

sync_start_PTS_of_PlayItem shown in Fig. 7 or 9) , which is 
the time on the Main Path, on which the IN point of the data 
referred to by the Sub Path is started in synchronization 
with the Man Path on the time axis of the Main Path. 
Accordingly, data (file) referred to by the Sub Path can be 
played back in synchronization with the main clip AV stream 
file referred to by the Main Path, as shown in Fig. 7 or 9 . 
[0220] 

Data read by the storage drive 31 shown in Fig. 2 5 may 
be data recorded on a recording medium, such as a DVD 
(Digital Versatile Disc) , data recorded on a hard disk, data 
downloaded via a network (not shown) , or data combined from 
such data. For example, data may be played back based on 
Playlist and a sub clip recorded on a hard disk and a main 
clip AV stream file recorded on a DVD. Alternatively, if 
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PlayList, using a clip AV stream file recorded on a DVD as a 
sub clip, and a main clip are recorded on a hard disk, the 
main clip and the sub clip may be read and played back from 
the hard disk and DVD, respectively, based on the PlayList 
recorded on the hard disk, 
[0221] 

The above -described series of processing operations may 
be executed by hardware or software. In this case, the 
processing operations can be performed by a personal 
computer 500 shown in Fig. 31. 
[0222] 

In Fig. 31, in the personal computer 500, a CPU 
(Central Processing Unit) 501 executes various processing 
operations according to a program stored in a ROM (Read Only 
Memory) 502 or a program loaded into a RAM (Random Access 
Memory) 503 from a storage unit 508. In the RAM 503, data 
necessary for the CPU 501 to execute various processing 
operations is also stored. 
[0223] 

The CPU 501, the ROM 502, and the RAM 503 are connected 
to each other via an internal bus 504. An input/output 
interface 505 is also connected to the internal bus 504. 
[0224] 

The input/output interface 505 is connected to an input 
unit 506, such as a keyboard and a mouse, an output unit 507, 
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such as a display, for example, a CRT or an LCD, the storage 
unit 508, such as a hard disk, and a communication unit 509, 
such as a modem or a terminal adapter. The communication 
unit 509 performs communication via various networks 
including telephone lines or CATV. 
[0225] 

A drive 510 is connected to the input/output interface 
5.05 if necessary. A removable medium 521, such as a 
magnetic disk, an optical disc, a magneto-optical disk, or a 
semiconductor memory, is installed in the drive 510. A 
computer program read from the removable medium 521 is 
installed into the storage unit 508. 
[0226] 

If software is used for executing the series of 
processing operations, a corresponding software program is 
installed from a network or a recording medium . 
[0227] 

This recording medium may be formed of a package medium, 
such as the removable medium 521, recording the program 
thereon, as shown in Fig. 31, which is distributed to the 
user separately from the computer. Alternatively, the 
recording medium may be formed of the ROM 502 or a hard disk 
forming the storage unit 508 recording the program thereon, 
which is distributed to the user while being built in the 
computer. 
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[0228] 

In this specification, steps forming the computer 
program may be executed in chronological order described in 
this specification. Alternatively, they may be executed in 
parallel or individually. 
[0229] 

In this specification, the system represents the 
overall apparatus including a plurality of devices. 



